{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# TensorBoard Usage -- Updated\n",
    "\n",
    "Looks like I need to relearn Tensorboard since my old code is all deprecated :/\n",
    "Tutorial link: [link](https://www.tensorflow.org/get_started/summaries_and_tensorboard)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "1. Create the TensorFlow graph that you'd like to collect summary data from, and decide which nodes you would like to annotate with summary operations.\n",
    "```python\n",
    "def variable_summaries(var):\n",
    "  \"\"\"Attach a lot of summaries to a Tensor (for TensorBoard visualization).\"\"\"\n",
    "  with tf.name_scope('summaries'):\n",
    "    mean = tf.reduce_mean(var)\n",
    "    tf.summary.scalar('mean', mean)\n",
    "    with tf.name_scope('stddev'):\n",
    "      stddev = tf.sqrt(tf.reduce_mean(tf.square(var - mean)))\n",
    "    tf.summary.scalar('stddev', stddev)\n",
    "    tf.summary.scalar('max', tf.reduce_max(var))\n",
    "    tf.summary.scalar('min', tf.reduce_min(var))\n",
    "    tf.summary.histogram('histogram', var)\n",
    "```\n",
    "\n",
    "2. To generate summaries, we need to run all of these summary nodes. Combine them into a single op that generates all the summary data.\n",
    "```python\n",
    "merged = tf.summary.merge_all()\n",
    "```\n",
    "\n",
    "3. Run the merged summary op, which will generate a serialized Summary protobuf object with all of your summary data at a given step. Finally, to write this summary data to disk, pass the summary protobuf to a tf.summary.FileWriter.\n",
    "```python\n",
    "train_writer = tf.summary.FileWriter(FLAGS.summaries_dir + '/train',\n",
    "                                      sess.graph)\n",
    "test_writer = tf.summary.FileWriter(FLAGS.summaries_dir + '/test')\n",
    "tf.global_variables_initializer().run()\n",
    "```\n",
    "\n",
    "4. During training . . . \n",
    "```python\n",
    "summary, acc = sess.run([merged, accuracy], feed_dict=feed_dict(False))\n",
    "    test_writer.add_summary(summary, i)\n",
    "```\n",
    "\n",
    "5. Launching tensorboard after training. \n",
    "```\n",
    "tensorboard --logdir=path/to/log-directory\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Description of below code (docstring):\n",
    "\n",
    "*A simple MNIST classifier which displays summaries in TensorBoard. This is an unimpressive MNIST model, but it is a good example of using tf.name\\_scope to make a graph legible in the TensorBoard graph explorer, and of naming summary tags so that they are grouped meaningfully in TensorBoard. It demonstrates the functionality of every TensorBoard dashboard.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "import argparse\n",
    "import sys\n",
    "import tensorflow as tf\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def train():\n",
    "    mnist = input_data.read_data_sets(FLAGS.data_dir, one_hot=True)\n",
    "\n",
    "    sess = tf.InteractiveSession()\n",
    "    # Create a multilayer model.\n",
    "\n",
    "    # Input placeholders\n",
    "    with tf.name_scope('input'):\n",
    "        x = tf.placeholder(tf.float32, [None, 784], name='x-input')\n",
    "        y_ = tf.placeholder(tf.float32, [None, 10], name='y-input')\n",
    "\n",
    "    with tf.name_scope('input_reshape'):\n",
    "        image_shaped_input = tf.reshape(x, [-1, 28, 28, 1])\n",
    "        tf.summary.image('input', image_shaped_input, 10)\n",
    "\n",
    "    # We can't initialize these variables to 0 - the network will get stuck.\n",
    "    def weight_variable(shape):\n",
    "        \"\"\"Create a weight variable with appropriate initialization.\"\"\"\n",
    "        initial = tf.truncated_normal(shape, stddev=0.1)\n",
    "        return tf.Variable(initial)\n",
    "\n",
    "    def bias_variable(shape):\n",
    "        \"\"\"Create a bias variable with appropriate initialization.\"\"\"\n",
    "        initial = tf.constant(0.1, shape=shape)\n",
    "        return tf.Variable(initial)\n",
    "\n",
    "    def variable_summaries(var):\n",
    "        \"\"\"Attach a lot of summaries to a Tensor (for TensorBoard visualization).\"\"\"\n",
    "        with tf.name_scope('summaries'):\n",
    "            mean = tf.reduce_mean(var)\n",
    "            tf.summary.scalar('mean', mean)\n",
    "            with tf.name_scope('stddev'):\n",
    "                stddev = tf.sqrt(tf.reduce_mean(tf.square(var - mean)))\n",
    "            tf.summary.scalar('stddev', stddev)\n",
    "            tf.summary.scalar('max', tf.reduce_max(var))\n",
    "            tf.summary.scalar('min', tf.reduce_min(var))\n",
    "            tf.summary.histogram('histogram', var)\n",
    "\n",
    "    def nn_layer(input_tensor, input_dim, output_dim, layer_name, act=tf.nn.relu):\n",
    "        \"\"\"Reusable code for making a simple neural net layer.\n",
    "\n",
    "        It does a matrix multiply, bias add, and then uses relu to nonlinearize.\n",
    "        It also sets up name scoping so that the resultant graph is easy to read,\n",
    "        and adds a number of summary ops.\n",
    "        \"\"\"\n",
    "        # Adding a name scope ensures logical grouping of the layers in the graph.\n",
    "        with tf.name_scope(layer_name):\n",
    "            # This Variable will hold the state of the weights for the layer\n",
    "            with tf.name_scope('weights'):\n",
    "                weights = weight_variable([input_dim, output_dim])\n",
    "                variable_summaries(weights)\n",
    "            with tf.name_scope('biases'):\n",
    "                biases = bias_variable([output_dim])\n",
    "                variable_summaries(biases)\n",
    "            with tf.name_scope('Wx_plus_b'):\n",
    "                preactivate = tf.matmul(input_tensor, weights) + biases\n",
    "                tf.summary.histogram('pre_activations', preactivate)\n",
    "            activations = act(preactivate, name='activation')\n",
    "            tf.summary.histogram('activations', activations)\n",
    "            return activations\n",
    "\n",
    "    hidden1 = nn_layer(x, 784, 500, 'layer1')\n",
    "\n",
    "    with tf.name_scope('dropout'):\n",
    "        keep_prob = tf.placeholder(tf.float32)\n",
    "        tf.summary.scalar('dropout_keep_probability', keep_prob)\n",
    "        dropped = tf.nn.dropout(hidden1, keep_prob)\n",
    "\n",
    "    # Do not apply softmax activation yet, see below.\n",
    "    y = nn_layer(dropped, 500, 10, 'layer2', act=tf.identity)\n",
    "\n",
    "    with tf.name_scope('cross_entropy'):\n",
    "        # The raw formulation of cross-entropy,\n",
    "        #\n",
    "        # tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(tf.softmax(y)),\n",
    "        #                               reduction_indices=[1]))\n",
    "        #\n",
    "        # can be numerically unstable.\n",
    "        #\n",
    "        # So here we use tf.nn.softmax_cross_entropy_with_logits on the\n",
    "        # raw outputs of the nn_layer above, and then average across\n",
    "        # the batch.\n",
    "        diff = tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y)\n",
    "        with tf.name_scope('total'):\n",
    "            cross_entropy = tf.reduce_mean(diff)\n",
    "    tf.summary.scalar('cross_entropy', cross_entropy)\n",
    "\n",
    "    with tf.name_scope('train'):\n",
    "        train_step = tf.train.AdamOptimizer(FLAGS.learning_rate).minimize(cross_entropy)\n",
    "\n",
    "    with tf.name_scope('accuracy'):\n",
    "        with tf.name_scope('correct_prediction'):\n",
    "            correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))\n",
    "        with tf.name_scope('accuracy'):\n",
    "            accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "    tf.summary.scalar('accuracy', accuracy)\n",
    "\n",
    "    # Merge all the summaries and write them out to /tmp/tensorflow/mnist/logs/mnist_with_summaries (by default)\n",
    "    merged = tf.summary.merge_all()\n",
    "    train_writer = tf.summary.FileWriter(FLAGS.log_dir + '/train', sess.graph)\n",
    "    test_writer = tf.summary.FileWriter(FLAGS.log_dir + '/test')\n",
    "    tf.global_variables_initializer().run()\n",
    "\n",
    "    # Train the model, and also write summaries.\n",
    "    # Every 10th step, measure test-set accuracy, and write test summaries\n",
    "    # All other steps, run train_step on training data, & add training summaries\n",
    "\n",
    "    def feed_dict(train):\n",
    "        \"\"\"Make a TensorFlow feed_dict: maps data onto Tensor placeholders.\"\"\"\n",
    "        if train or FLAGS.fake_data:\n",
    "            xs, ys = mnist.train.next_batch(100, fake_data=FLAGS.fake_data)\n",
    "            k = FLAGS.dropout\n",
    "        else:\n",
    "            xs, ys = mnist.test.images, mnist.test.labels\n",
    "            k = 1.0\n",
    "        return {x: xs, y_: ys, keep_prob: k}\n",
    "\n",
    "    for i in range(FLAGS.max_steps):\n",
    "        if i % 10 == 0:  # Record summaries and test-set accuracy\n",
    "            summary, acc = sess.run([merged, accuracy], feed_dict=feed_dict(False))\n",
    "            test_writer.add_summary(summary, i)\n",
    "            print('Accuracy at step %s: %s' % (i, acc))\n",
    "        else:  # Record train set summaries, and train\n",
    "            if i % 100 == 99:  # Record execution stats\n",
    "                run_options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)\n",
    "                run_metadata = tf.RunMetadata()\n",
    "                summary, _ = sess.run([merged, train_step],\n",
    "                                      feed_dict=feed_dict(True),\n",
    "                                      options=run_options,\n",
    "                                      run_metadata=run_metadata)\n",
    "                train_writer.add_run_metadata(run_metadata, 'step%03d' % i)\n",
    "                train_writer.add_summary(summary, i)\n",
    "                print('Adding run metadata for', i)\n",
    "            else:  # Record a summary\n",
    "                summary, _ = sess.run([merged, train_step], feed_dict=feed_dict(True))\n",
    "                train_writer.add_summary(summary, i)\n",
    "    train_writer.close()\n",
    "    test_writer.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def main(_):\n",
    "    if tf.gfile.Exists(FLAGS.log_dir):\n",
    "        tf.gfile.DeleteRecursively(FLAGS.log_dir)\n",
    "    tf.gfile.MakeDirs(FLAGS.log_dir)\n",
    "    train()\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    parser = argparse.ArgumentParser()\n",
    "    parser.add_argument('--fake_data', nargs='?', const=True, type=bool,\n",
    "              default=False,\n",
    "              help='If true, uses fake data for unit testing.')\n",
    "    parser.add_argument('--max_steps', type=int, default=1000,\n",
    "              help='Number of steps to run trainer.')\n",
    "    parser.add_argument('--learning_rate', type=float, default=0.001,\n",
    "              help='Initial learning rate')\n",
    "    parser.add_argument('--dropout', type=float, default=0.9,\n",
    "              help='Keep probability for training dropout.')\n",
    "    parser.add_argument('--data_dir', type=str, default='/tmp/tensorflow/mnist/input_data',\n",
    "                      help='Directory for storing input data')\n",
    "    parser.add_argument('--log_dir', type=str, default='/tmp/tensorflow/mnist/logs/mnist_with_summaries',help='Summaries log directory')\n",
    "    FLAGS, unparsed = parser.parse_known_args()\n",
    "    tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Trying to save the Graph Again"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "[Tutorial Link](https://www.tensorflow.org/programmers_guide/meta_graph)\n",
    "\n",
    "In order for a Python object to be serialized to and from MetaGraphDef, the Python class must implement to_proto() and from_proto() methods, and register them with the system using register_proto_function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "class Variable:\n",
    "\n",
    "    def to_proto(self, export_scope=None):\n",
    "        \"\"\"Converts a `Variable` to a `VariableDef` protocol buffer.\n",
    "\n",
    "        Args:\n",
    "        export_scope: Optional `string`. Name scope to remove.\n",
    "\n",
    "        Returns:\n",
    "        A `VariableDef` protocol buffer, or `None` if the `Variable` is not\n",
    "        in the specified name scope.\n",
    "        \"\"\"\n",
    "        if (export_scope is None or self._variable.name.startswith(export_scope)):\n",
    "            var_def                  = variable_pb2.VariableDef()\n",
    "            var_def.variable_name    = ops.strip_name_scope(self._variable.name, export_scope)\n",
    "            var_def.initializer_name = ops.strip_name_scope(self.initializer.name, export_scope)\n",
    "            var_def.snapshot_name    = ops.strip_name_scope(self._snapshot.name, export_scope)\n",
    "            if self._save_slice_info:\n",
    "                var_def.save_slice_info_def.MergeFrom(self._save_slice_info.to_proto(export_scope=export_scope))\n",
    "            return var_def\n",
    "        else:\n",
    "            return None\n",
    "\n",
    "    @staticmethod\n",
    "    def from_proto(variable_def, import_scope=None):\n",
    "        \"\"\"Returns a `Variable` object created from `variable_def`.\"\"\"\n",
    "        return Variable(variable_def=variable_def, import_scope=import_scope)\n",
    "\n",
    "\n",
    "ops.register_proto_function(ops.GraphKeys.GLOBAL_VARIABLES,\n",
    "                            proto_type=variable_pb2.VariableDef,\n",
    "                            to_proto=Variable.to_proto,\n",
    "                            from_proto=Variable.from_proto)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "## Exporting a MetaGraph\n",
    "\n",
    "The function signature, for reference:\n",
    "```python\n",
    "def export_meta_graph(filename=None, collection_list=None, as_text=False):\n",
    "  \"\"\"Writes `MetaGraphDef` to save_path/filename.\n",
    "\n",
    "  Args:\n",
    "    filename: Optional meta_graph filename including the path.\n",
    "    collection_list: List of string keys to collect.\n",
    "    as_text: If `True`, writes the meta_graph as an ASCII proto.\n",
    "\n",
    "  Returns:\n",
    "    A `MetaGraphDef` proto.\n",
    "  \"\"\"\n",
    "```\n",
    "\n",
    "You should store all variables you wanna get letter in ```collection```. Or, if you don't specify one, it will save everything (yeah do that). \n",
    "\n",
    "How to export the default running graph:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "logits = tf.matmul(hidden2, weights) + biases\n",
    "tf.add_to_collection(\"logits\", logits)\n",
    "\n",
    "init_all_op = tf.global_variables_initializer()\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    # Initializes all the variables.\n",
    "    sess.run(init_all_op)\n",
    "    # Runs to logit.\n",
    "    sess.run(logits)\n",
    "    chatbot.saver.save(sess, 'my-save-dir/my-model-10000')\n",
    "    # Generates MetaGraphDef.\n",
    "    chatbot.saver.export_meta_graph('my-save-dir/my-model-10000.meta')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "## Importing the MetaGraph and Resuming Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### Case: Import and Extend Graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "with tf.Session() as sess:\n",
    "    new_saver = tf.train.import_meta_graph('my-save-dir/my-model-10000.meta')\n",
    "    new_saver.restore(sess, 'my-save-dir/my-model-10000')\n",
    "    \n",
    "    # Adds loss and train.\n",
    "    labels = tf.constant(0, tf.int32, shape=[100], name=\"labels\")\n",
    "    batch_size = tf.size(labels)\n",
    "    labels = tf.expand_dims(labels, 1)\n",
    "    indices = tf.expand_dims(tf.range(0, batch_size), 1)\n",
    "    concated = tf.concat([indices, labels], 1)\n",
    "    onehot_labels = tf.sparse_to_dense(concated, tf.stack([batch_size, 10]), 1.0, 0.0)\n",
    "    \n",
    "    # Getting the logits variable back that we saved.\n",
    "    logits = tf.get_collection(\"logits\")[0]\n",
    "    cross_entropy = tf.nn.softmax_cross_entropy_with_logits(labels=onehot_labels, logits=logits, name=\"xentropy\")\n",
    "    loss = tf.reduce_mean(cross_entropy, name=\"xentropy_mean\")\n",
    "\n",
    "    tf.summary.scalar('loss', loss)\n",
    "    # Creates the gradient descent optimizer with the given learning rate.\n",
    "    optimizer = tf.train.GradientDescentOptimizer(0.01)\n",
    "\n",
    "    # Runs train_op.\n",
    "    train_op = optimizer.minimize(loss)\n",
    "    sess.run(train_op)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### Case: Retrieve Hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "filename = \".\".join([tf.latest_checkpoint(train_dir), \"meta\"])\n",
    "tf.train.import_meta_graph(filename)\n",
    "hparams = tf.get_collection(\"hparams\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Sharing Variables\n",
    "\n",
    "My version of TensorFlow's tutorial by the same name, but in English. The (sub)section names are identical; I compress the content. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "## The Problem\n",
    "\n",
    "**Reusing the same model on multiple inputs**: Let's say we make a function ```my_model(inputs)``` that creates a bunch of tf.Variable() objects and returns the final output tensor variable. Each time we call my_model(next_inputs), it is going to create a whole new model with new variables. This bad. \n",
    "\n",
    "**How to fix?**: Rather than requiring classes to create a model and manage the variables, TensorFlow provides a *Variable scope* mechanism for sharing named variables while constructing a graph."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "## Variable Scope\n",
    "\n",
    "The mechanism consists of 2 main functions:\n",
    "1. ```tf.get_variable(name, shape, initializer)``` creates or returns a variable with a given name.\n",
    "2. ```tf.variable_scope(scope_name)``` manages namespaces **for names passed to tf.get_variable()**. \n",
    "\n",
    "\n",
    "__Difference between name scope and variable scope__: [Here is a link](http://stackoverflow.com/questions/35919020/whats-the-difference-of-name-scope-and-a-variable-scope-in-tensorflow) to the best explanation I could find. The following is the main part:\n",
    "\n",
    "\"However, name scope is ignored by tf.get_variable. We can see that in the following example:\n",
    "\n",
    "```python\n",
    "with tf.name_scope(\"my_scope\"):\n",
    "    v1 = tf.get_variable(\"var1\", [1], dtype=tf.float32)\n",
    "    v2 = tf.Variable(1, name=\"var2\", dtype=tf.float32)\n",
    "    a = tf.add(v1, v2)\n",
    "\n",
    "print(v1.name)  # var1:0\n",
    "print(v2.name)  # my_scope/var2:0\n",
    "print(a.name)   # my_scope/Add:0\n",
    "```\n",
    "\n",
    "The only way to place a variable accessed using tf.get_variable in a scope is to use variable scope, as in the following example:\n",
    "\n",
    "```python\n",
    "with tf.variable_scope(\"my_scope\"):\n",
    "    v1 = tf.get_variable(\"var1\", [1], dtype=tf.float32)\n",
    "    v2 = tf.Variable(1, name=\"var2\", dtype=tf.float32)\n",
    "    a = tf.add(v1, v2)\n",
    "\n",
    "print(v1.name)  # my_scope/var1:0\n",
    "print(v2.name)  # my_scope/var2:0\n",
    "print(a.name)   # my_scope/Add:0\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RNN Documentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RNNCell"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interface Specifications\n",
    "\n",
    "It's extremely helpful to read the [source code for RNNCell](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/rnn_cell_impl.py). Let's see a condensed list of the important methods and descriptions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "* __\\_\\_call\\_\\___: Run this RNN cell on inputs, starting from the given state.\n",
    "\n",
    "```python\n",
    "def __call__(self, inputs, state, scope=None):\n",
    "  \"\"\"Args:\n",
    "      inputs: `2-D` tensor with shape `[batch_size x input_size]`.\n",
    "      state: if `self.state_size` is an integer, this should be a `2-D Tensor`\n",
    "        with shape `[batch_size x self.state_size]`.  Otherwise, if\n",
    "        `self.state_size` is a tuple of integers, this should be a tuple\n",
    "        with shapes `[batch_size x s] for s in self.state_size`.\n",
    "      scope: VariableScope for the created subgraph; defaults to class name.\n",
    "    Returns:\n",
    "      A pair containing:\n",
    "      - Output: A `2-D` tensor with shape `[batch_size x self.output_size]`.\n",
    "      - New state: Either a single `2-D` tensor, or a tuple of tensors matching\n",
    "        the arity and shapes of `state`.\n",
    "    \"\"\"\n",
    "```\n",
    "\n",
    "* __state\\_size and output\\_size__:\n",
    "\n",
    "```python\n",
    "  @property\n",
    "  def state_size(self):\n",
    "    \"\"\"size(s) of state(s) used by this cell.\n",
    "    It can be represented by an Integer, a TensorShape or a tuple of Integers\n",
    "    or TensorShapes.\n",
    "    \"\"\"\n",
    "\n",
    "  @property\n",
    "  def output_size(self):\n",
    "    \"\"\"Integer or TensorShape: size of outputs produced by this cell.\"\"\"\n",
    "```\n",
    "\n",
    "* __zero\\_state__:\n",
    "```python\n",
    "  def zero_state(self, batch_size, dtype):\n",
    "    \"\"\"Return zero-filled state tensor(s).\n",
    "    Args:\n",
    "      batch_size: int, float, or unit Tensor representing the batch size.\n",
    "      dtype: the data type to use for the state.\n",
    "    Returns:\n",
    "      If `state_size` is an int or TensorShape, then the return value is a\n",
    "      `N-D` tensor of shape `[batch_size x state_size]` filled with zeros.\n",
    "      If `state_size` is a nested list or tuple, then the return value is\n",
    "      a nested list or tuple (of the same structure) of `2-D` tensors with\n",
    "      the shapes `[batch_size x s]` for each s in `state_size`.\n",
    "    \"\"\"\n",
    "    with ops.name_scope(type(self).__name__ + \"ZeroState\", values=[batch_size]):\n",
    "      state_size = self.state_size\n",
    "      return _zero_state_tensors(state_size, batch_size, dtype)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decoder Class -- Currently undocumented . . . "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here I'm going to give myself an overview of this strange new set of files in tf.contrib.seq2seq (master branch) that is not on the website. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basic Decoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Miscellaneous"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Review: logits terminology\n",
    "\n",
    "1.  Logistic function: \n",
    "$$\\begin{align}\n",
    "p(X) &= \\frac{1}{1 + e^{-\\beta^TX}}  \\\\\n",
    "&= \\frac{e^{ \\beta^TX}}{1 + e^{ \\beta^TX}}\n",
    "\\end{align}$$\n",
    "\n",
    "2. Odds: \n",
    "$$\n",
    "\\frac{p(X)}{1 - p(X)} = e^{\\beta^T X}\n",
    "$$\n",
    "\n",
    "3. log-odds, \"logit\":\n",
    "$$\n",
    "\\log\\left(\\frac{p(X)}{1 - p(X)} \\right) = \\beta^T X\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Random Facts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* __tf.Variable vs tf.placeholder__: \"The difference is that with tf.Variable you have to provide an initial value when you declare it. With tf.placeholder you don't have to provide an initial value and you can specify it at run time with the feed_dict argument inside Session.run\" [Source](http://stackoverflow.com/questions/36693740/whats-the-difference-between-tf-placeholder-and-tf-variable)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Embedding (All Related Topics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Embedding Lookup\n",
    "\n",
    "Goal: Figure out how embedding lookup works. Ideally i'd like to show, as a toy example, an integer being mapped to its binary string. For example, with embed size of 5\n",
    "```python\n",
    "embed([2, 19]) --> [[0, 0, 0, 1, 0], [1, 0, 0, 1, 1]] \n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "\n",
    "input_size = (1, 2) # arbitrary\n",
    "embed_size = 5\n",
    "vocab_size = 2**embed_size\n",
    "input_array = np.random.randint(vocab_size, size=input_size)\n",
    "expected_embedding = ['{0:b}'.format(inp) for inp in input_array.flatten()]\n",
    "print(\"Input array:\\n\", input_array)\n",
    "print(\"Expected embedding:\\n\", expected_embedding)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[[ 0.30149388 -0.099668    0.14693063 -0.39335564  0.34846574]\n",
      "  [-0.0505532  -0.21680753 -0.3851763  -0.1110653   0.0726169 ]]]\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "sess = None\n",
    "with tf.Session() as sess:\n",
    "    params   = tf.get_variable('embed_tensor', \n",
    "                               shape=[vocab_size, embed_size], \n",
    "                               initializer=tf.contrib.layers.xavier_initializer())\n",
    "    inputs    = tf.convert_to_tensor(input_array)\n",
    "    embedding = tf.nn.embedding_lookup(params, inputs)\n",
    "    \n",
    "    \n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    emb = sess.run(embedding)\n",
    "    print(emb)\n",
    "\n",
    "sess = None\n",
    "tf.reset_default_graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Embedding Visualizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from __future__ import absolute_import\n",
    "from __future__ import division\n",
    "from __future__ import print_function\n",
    "import argparse\n",
    "import sys \n",
    "import os\n",
    "import numpy as np\n",
    "from tensorflow.contrib.tensorboard.plugins import projector\n",
    "import tensorflow as tf\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "FLAGS = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def generate_embeddings():\n",
    "    # Import data\n",
    "    mnist = input_data.read_data_sets(FLAGS.data_dir, one_hot=True, fake_data=FLAGS.fake_data)\n",
    "    sess = tf.InteractiveSession()\n",
    "    # Input set for Embedded TensorBoard visualization\n",
    "    # Performed with cpu to conserve memory and processing power\n",
    "    with tf.device(\"/cpu:0\"):\n",
    "        embedding = tf.Variable(tf.stack(mnist.test.images[:FLAGS.max_steps], axis=0), \n",
    "                                trainable=False, name='embedding')\n",
    "    tf.global_variables_initializer().run()\n",
    "    saver = tf.train.Saver()\n",
    "    writer = tf.summary.FileWriter(FLAGS.log_dir + '/projector', sess.graph)\n",
    "    # Add embedding tensorboard visualization. \n",
    "    config = projector.ProjectorConfig()\n",
    "    embed= config.embeddings.add()\n",
    "    embed.tensor_name = 'embedding:0'\n",
    "    embed.metadata_path = os.path.join(FLAGS.log_dir + '/projector/metadata.tsv')\n",
    "    embed.sprite.image_path = os.path.join(FLAGS.data_dir + '/mnist_10k_sprite.png')\n",
    "    # Specify the width and height of a single thumbnail.\n",
    "    embed.sprite.single_image_dim.extend([28, 28])\n",
    "    projector.visualize_embeddings(writer, config)\n",
    "    saver.save(sess, os.path.join(\n",
    "        FLAGS.log_dir, 'projector/a_model.ckpt'), global_step=FLAGS.max_steps)\n",
    "\n",
    "def generate_metadata_file():\n",
    "    # Import data\n",
    "    mnist = input_data.read_data_sets(FLAGS.data_dir, one_hot=True, fake_data=FLAGS.fake_data)\n",
    "    def save_metadata(file):\n",
    "        with open(file, 'w') as f:\n",
    "            for i in range(FLAGS.max_steps):\n",
    "                c = np.nonzero(mnist.test.labels[::1])[1:][0][i]\n",
    "                f.write('{}\\n'.format(c))\n",
    "    save_metadata(FLAGS.log_dir + '/projector/metadata.tsv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def main(_):\n",
    "    if tf.gfile.Exists(FLAGS.log_dir + '/projector'):\n",
    "        tf.gfile.DeleteRecursively(FLAGS.log_dir + '/projector')\n",
    "        tf.gfile.MkDir(FLAGS.log_dir + '/projector')\n",
    "    tf.gfile.MakeDirs(FLAGS.log_dir  + '/projector') # fix the directory to be created\n",
    "    generate_metadata_file()\n",
    "    generate_embeddings()\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    BASE = '/home/brandon/mnist-tensorboard-embeddings/'\n",
    "    parser = argparse.ArgumentParser()\n",
    "    parser.add_argument('--fake_data', nargs='?', const=True, type=bool,\n",
    "                        default=False, help='If true, uses fake data for unit testing.')\n",
    "    parser.add_argument('--max_steps', type=int, default=10000, help='Number of steps to run trainer.')\n",
    "    parser.add_argument('--data_dir', type=str, default=BASE+'mnist_data', help='Directory for storing input data')\n",
    "    parser.add_argument('--log_dir', type=str, default=BASE+'logs', help='Summaries log directory')\n",
    "    FLAGS, unparsed = parser.parse_known_args()\n",
    "    tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### YES: TensorBoard Summit Tutorial Code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting /tmp/mnist_tutorial/data/train-images-idx3-ubyte.gz\n",
      "Extracting /tmp/mnist_tutorial/data/train-labels-idx1-ubyte.gz\n",
      "Extracting /tmp/mnist_tutorial/data/t10k-images-idx3-ubyte.gz\n",
      "Extracting /tmp/mnist_tutorial/data/t10k-labels-idx1-ubyte.gz\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "('/tmp/mnist_tutorial/sprite_1024.png',\n",
       " <http.client.HTTPMessage at 0x7f59ff6014e0>)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import tensorflow as tf\n",
    "import urllib\n",
    "import urllib.request\n",
    "\n",
    "LOGDIR = '/tmp/mnist_tutorial/'\n",
    "GIST_URL = 'https://gist.githubusercontent.com/dandelionmane/4f02ab8f1451e276fea1f165a20336f1/raw/                            dfb8ee95b010480d56a73f324aca480b3820c180'\n",
    "\n",
    "### MNIST EMBEDDINGS ###\n",
    "mnist = tf.contrib.learn.datasets.mnist.read_data_sets(train_dir=LOGDIR + 'data', one_hot=True)\n",
    "### Get a sprite and labels file for the embedding projector ###\n",
    "urllib.request.urlretrieve(GIST_URL + 'labels_1024.tsv', LOGDIR + 'labels_1024.tsv')\n",
    "urllib.request.urlretrieve(GIST_URL + 'sprite_1024.png', LOGDIR + 'sprite_1024.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def conv_layer(input, size_in, size_out, name=\"conv\"):\n",
    "    with tf.name_scope(name):\n",
    "        w = tf.Variable(tf.truncated_normal([5, 5, size_in, size_out], stddev=0.1), name=\"W\")\n",
    "        b = tf.Variable(tf.constant(0.1, shape=[size_out]), name=\"B\")\n",
    "        conv = tf.nn.conv2d(input, w, strides=[1, 1, 1, 1], padding=\"SAME\")\n",
    "        act = tf.nn.relu(conv + b)\n",
    "        tf.summary.histogram(\"weights\", w)\n",
    "        tf.summary.histogram(\"biases\", b)\n",
    "        tf.summary.histogram(\"activations\", act)\n",
    "        return tf.nn.max_pool(act, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding=\"SAME\")\n",
    "    \n",
    "def fc_layer(input, size_in, size_out, name=\"fc\"):\n",
    "    with tf.name_scope(name):\n",
    "        w = tf.Variable(tf.truncated_normal([size_in, size_out], stddev=0.1), name=\"W\")\n",
    "        b = tf.Variable(tf.constant(0.1, shape=[size_out]), name=\"B\")\n",
    "        act = tf.nn.relu(tf.matmul(input, w) + b)\n",
    "        tf.summary.histogram(\"weights\", w)\n",
    "        tf.summary.histogram(\"biases\", b)\n",
    "        tf.summary.histogram(\"activations\", act)\n",
    "        return act "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def mnist_model(learning_rate, use_two_conv, use_two_fc, hparam):\n",
    "    tf.reset_default_graph()\n",
    "    sess = tf.Session()\n",
    "    # Setup placeholders, and reshape the data\n",
    "    x = tf.placeholder(tf.float32, shape=[None, 784], name=\"x\")\n",
    "    x_image = tf.reshape(x, [-1, 28, 28, 1]) \n",
    "    tf.summary.image('input', x_image, 3)\n",
    "    y = tf.placeholder(tf.float32, shape=[None, 10], name=\"labels\")\n",
    "    if use_two_conv:\n",
    "        conv1 = conv_layer(x_image, 1, 32, \"conv1\")\n",
    "        conv_out = conv_layer(conv1, 32, 64, \"conv2\")\n",
    "    else:\n",
    "        conv1 = conv_layer(x_image, 1, 64, \"conv\")\n",
    "        conv_out = tf.nn.max_pool(conv1, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding=\"SAME\")\n",
    "    flattened = tf.reshape(conv_out, [-1, 7 * 7 * 64])\n",
    "    if use_two_fc:\n",
    "        fc1 = fc_layer(flattened, 7 * 7 * 64, 1024, \"fc1\")\n",
    "        embedding_input = fc1 \n",
    "        embedding_size = 1024\n",
    "        logits = fc_layer(fc1, 1024, 10, \"fc2\")\n",
    "    else:\n",
    "        embedding_input = flattened\n",
    "        embedding_size = 7*7*64\n",
    "        logits = fc_layer(flattened, 7*7*64, 10, \"fc\")\n",
    "    with tf.name_scope(\"xent\"):\n",
    "        xent = tf.reduce_mean(\n",
    "        tf.nn.softmax_cross_entropy_with_logits(\n",
    "            logits=logits, labels=y), name=\"xent\")\n",
    "        tf.summary.scalar(\"xent\", xent)\n",
    "    with tf.name_scope(\"train\"):\n",
    "        train_step = tf.train.AdamOptimizer(learning_rate).minimize(xent)\n",
    "    with tf.name_scope(\"accuracy\"):\n",
    "        correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(y, 1)) \n",
    "        accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "        tf.summary.scalar(\"accuracy\", accuracy)\n",
    "    summ = tf.summary.merge_all()\n",
    "    \n",
    "    # _________ EMBEDDING VARIABLE/TENSOR CREATED _________\n",
    "    embedding = tf.Variable(tf.zeros([1024, embedding_size]), name=\"test_embedding\")\n",
    "    assignment = embedding.assign(embedding_input)\n",
    "    \n",
    "    # _________ SAVER CREATED _________\n",
    "    saver = tf.train.Saver()\n",
    "    \n",
    "    # _________ INIT GLOBAL VARS IS RUN  _________\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    \n",
    "    # _________ WRITER IS CREATED  _________\n",
    "    writer = tf.summary.FileWriter(LOGDIR + hparam)\n",
    "    writer.add_graph(sess.graph)\n",
    "    \n",
    "    # ================================================================\n",
    "    # _________ PROJECTORCONFIG CONSTRUCTOR  _________\n",
    "    config = tf.contrib.tensorboard.plugins.projector.ProjectorConfig()\n",
    "    embedding_config = config.embeddings.add()\n",
    "    embedding_config.tensor_name = embedding.name\n",
    "    embedding_config.sprite.image_path = LOGDIR + 'sprite_1024.png'\n",
    "    embedding_config.metadata_path = LOGDIR + 'labels_1024.tsv'\n",
    "    # Specify the width and height of a single thumbnail.\n",
    "    embedding_config.sprite.single_image_dim.extend([28, 28])\n",
    "    tf.contrib.tensorboard.plugins.projector.visualize_embeddings(writer, config)\n",
    "    # ================================================================\n",
    "    \n",
    "    for i in range(2001):\n",
    "        batch = mnist.train.next_batch(100)\n",
    "        if i % 5 == 0:\n",
    "            [train_accuracy, s] = sess.run([accuracy, summ], feed_dict={x: batch[0], y: batch[1]})\n",
    "            writer.add_summary(s, i)\n",
    "        if i % 500 == 0:\n",
    "            sess.run(assignment, feed_dict={x: mnist.test.images[:1024], y: mnist.test.labels[:1024]})\n",
    "            saver.save(sess, os.path.join(LOGDIR, \"model.ckpt\"), i)\n",
    "        sess.run(train_step, feed_dict={x: batch[0], y: batch[1]})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def make_hparam_string(learning_rate, use_two_fc, use_two_conv):\n",
    "    conv_param = \"conv=2\" if use_two_conv else \"conv=1\"\n",
    "    fc_param = \"fc=2\" if use_two_fc else \"fc=1\"\n",
    "    return \"lr_%.0E,%s,%s\" % (learning_rate, conv_param, fc_param)\n",
    "\n",
    "def main():\n",
    "    # You can try adding some more learning rates\n",
    "    for learning_rate in [1E-4]:\n",
    "        # Include \"False\" as a value to try different model architectures\n",
    "        for use_two_fc in [True]:\n",
    "            for use_two_conv in [True]:\n",
    "                # Construct a hyperparameter string for each one (example: \"lr_1E-3,fc=2,conv=2)\n",
    "                hparam = make_hparam_string(learning_rate, use_two_fc, use_two_conv)\n",
    "                print('Starting run for %s' % hparam)\n",
    "\n",
    "                # Actually run with the new settings\n",
    "                mnist_model(learning_rate, use_two_fc, use_two_conv, hparam)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Misc. Ops Testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "tf concat with -1 as an argument? No documentation on it so guess we'll have to try it out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "\n",
    "a = tf.convert_to_tensor(np.arange(20).reshape(5, 4))\n",
    "concat = tf.concat([[-1, 1], tf.shape(a)], 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-1  1  5  4]\n"
     ]
    }
   ],
   "source": [
    "# Config (hopefully) fixes GPU not releasing memory issue in jupyter.\n",
    "config = tf.ConfigProto()\n",
    "config.gpu_options.allow_growth=True\n",
    "with tf.Session(config=config) as sess:\n",
    "    print(sess.run(concat))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
